A mathematical analysis was carried out theoretically to reveal the fact that the Firefly Algorithm (FA) gets the risk of premature convergence and being trapped in local optimum. A hybrid Memetic algorithm based on simulated annealing was proposed. In the hybrid algorithm, the FA was employed to keep the diversity of firefly population and global exploration ability of the proposed algorithm. And then, the simulated annealing operator was incorporated to get rid of local optimum, which was utilized to carry out local search with partial firefly individuals by accepting bad solutions with some probability, and the proposed algorithm conducted simultaneously the attracting process and the annealing process to reduce the complexity. Finally, the performance of the proposed algorithm and other comparison algorithms were tested on ten standard functions, respectively. The experimental results show that the proposed algorithm can find the optimal solutions in six functions, outperform firefly algorithm, particle swarm optimization, etc, in terms of optimal value, mean value and standard deviation, and find better solutions than firefly algorithm in four functions.